﻿Exploratory data analysis (EDA) is an approach to analysing data for the purpose of formulating hypotheses worth testing, complementing the tools of conventional statistics for testing hypotheses.[1] It was so named by John Tukey to contrast with Confirmatory Data Analysis, the term used for the set of ideas about hypothesis testing, p-values, confidence intervals etc. which formed the key tools in the arsenal of practising statisticians at the time.

Contents [hide]
1 EDA development 
2 Techniques 
3 History 
4 Software 
5 See also 
6 References 
7 Bibliography 
8 External links 
 
[edit] EDA developmentTukey held that too much emphasis in statistics was placed on statistical hypothesis testing (confirmatory data analysis); more emphasis needed to be placed on using data to suggest hypotheses to test. In particular, he held that confusing the two types of analyses and employing them on the same set of data can lead to systematic bias owing to the issues inherent in testing hypotheses suggested by the data.

The objectives of EDA are to:

Suggest hypotheses about the causes of observed phenomena 
Assess assumptions on which statistical inference will be based 
Support the selection of appropriate statistical tools and techniques 
Provide a basis for further data collection through surveys or experiments 
Many EDA techniques have been adopted into data mining and are being taught to young students as a way to introduce them to statistical thinking.[2]

[edit] TechniquesThere are a number of tools that are useful for EDA, but EDA is characterized more by the attitude taken than by particular techniques.[3]

The principal graphical techniques used in EDA are:

Box plot 
Histogram 
Multi-vari chart 
Run chart 
Pareto chart 
Scatter plot 
Stem-and-leaf plot 
Odds ratio 
Chi-square 
The principal quantitative techniques are:

Median polish 
the Trimean 
Letter values 
Resistant line 
Resistant smooth 
Rootogram 
Graphical and quantitative techniques are:

Multidimensional scaling 
Ordination 
[edit] HistoryMany EDA ideas can be traced back to earlier authors, for example:

Francis Galton emphasized order statistics and quantiles. 
Arthur Bowley used precursors of the stemplot and five-number summary (Bowley actually used a "seven-figure summary", including the extremes, deciles and quartiles, along with the median - see his Elementary Manual of Statistics (3rd edn., 1920), p. 62 - he defines "the maximum and minimum, median, quartiles and two deciles" as the "seven positions"). 
Andrew Ehrenberg articulated a philosophy of data reduction (see his book of the same name). 
The Open University course Statistics in Society (MDST 242), took the above ideas and merged them with Gottfried Noether's work, which introduced statistical inference via coin-tossing and the median test.

[edit] SoftwareOpenSHAPA (modern open source successor to MacSHAPA), permits analysis of various media files (e.g. video, sound). 
CMU-DAP (Carnegie-Mellon University Data Analysis Package, FORTRAN source for EDA tools with English-style command syntax, 1977). 
Data Applied, a comprehensive web-based data visualization and data mining environment. 
Fathom (for high-school and intro college courses). 
JMP, an EDA package from SAS Institute. 
KNIME Konstanz Information Miner - Open-Source data exploration platform based on Eclipse. 
LiveGraph (open source real-time data series plotter). 
Orange, an open-source data mining software suite. 
SOCR provides a large number of free Internet-accessible. 
DASS-GUI - data mining framework written in C++ and Qt. 
TinkerPlots (for upper elementary and middle school students). 
[edit] See also Wikiversity has learning materials about Exploratory data analysis 
Anscombe's quartet, on importance of exploration 
Predictive analytics 
Structured data analysis (statistics) 
Data visualization is the study of the visual representation of data, meaning "information which has been abstracted in some schematic form, including attributes or variables for the units of information".[1]

According to Friedman (2008) the "main goal of data visualization is to communicate information clearly and effectively through graphical means. It doesn’t mean that data visualization needs to look boring to be functional or extremely sophisticated to look beautiful. To convey ideas effectively, both aesthetic form and functionality need to go hand in hand, providing insights into a rather sparse and complex data set by communicating its key-aspects in a more intuitive way. Yet designers often fail to achieve a balance between design and function, creating gorgeous data visualizations which fail to serve their main purpose — to communicate information".[2]

Data visualization is closely related to Information graphics, Information visualization, Scientific visualization and Statistical graphics. In the new millennium data visualization has become active area of research, teaching and development. According to Post et al (2002) it has united the field of scientific and information visualization".[3]

KPI Library has developed the “Periodic Table of Visualization Methods”, an interactive chart displaying various different data visualization methods [1]. It details 6 types of data visualization methods: data, information, concept, strategy, metaphor and compound.[citation needed]

Data visualization strategies were recently used by the Barack Obama administration as a way to enhance and broaden the support of his campaign. The "Road to Recovery" visualization became famous by showing US job loss figures between December 2007 and January 2010. This enabled people to compare the number of jobs lost during President Obama’s first year in office with the number of jobs lost during President Bush’s last year in office.[citation needed]

Contents [hide]
1 Data visualization scope 
2 Related fields 
2.1 Data acquisition 
2.2 Data analysis 
2.3 Data governance 
2.4 Data management 
2.5 Data mining 
3 See also 
4 References 
5 Further reading 
6 External links 
 
[edit] Data visualization scopeThere are different approaches on the scope of data visualization. One common focus is on information presentation such as Friedman (2008) presented it. On this way Friendly (2008) presumes two main parts of data visualization: statistical graphics, and thematic cartography.[1] In this line the "Data Visualization: Modern Approaches" (2007) article gives an overview of seven subjects of data visualization:[4]

Mindmaps 
Displaying news 
Displaying data 
Displaying connections 
Displaying websites 
Articles & resources 
Tools and services 
All these subjects are all close related to graphic design and information representation.

On the other hand, from a computer science perspective, Frits H. Post (2002) categorized the field into a number of sub-fields:[3]

Visualization algorithms and techniques 
Volume visualization 
Information visualization 
Multiresolution methods 
Modelling techniques and 
Interaction techniques and architectures 
[edit] Related fields[edit] Data acquisitionData acquisition is the sampling of the real world to generate data that can be manipulated by a computer. Sometimes abbreviated DAQ or DAS, data acquisition typically involves acquisition of signals and waveforms and processing the signals to obtain desired information. The components of data acquisition systems include appropriate sensors that convert any measurement parameter to an electrical signal, which is acquired by data acquisition hardware.

[edit] Data analysisData analysis is the process of looking at and summarizing data with the intent to extract useful information and develop conclusions. Data analysis is closely related to data mining, but data mining tends to focus on larger data sets, with less emphasis on making inference, and often uses data that was originally collected for a different purpose. In statistical applications, some people divide data analysis into descriptive statistics, exploratory data analysis and inferential statistics (or confirmatory data analysis), where the EDA focuses on discovering new features in the data, and CDA on confirming or falsifying existing hypotheses.

Types of data analysis are:

Exploratory data analysis (EDA): an approach to analyzing data for the purpose of formulating hypotheses worth testing, complementing the tools of conventional statistics for testing hypotheses. It was so named by John Tukey. 
Qualitative data analysis (QDA) or qualitative research is the analysis of non-numerical data, for example words, photographs, observations, etc.. 
[edit] Data governanceThis is very useful information for DM Seminars. Data governance encompasses the people, processes and technology required to create a consistent, enterprise view of an organisation's data in order to:

Increase consistency & confidence in decision making 
Decrease the risk of regulatory fines 
Improve data security 
Maximize the income generation potential of data 
Designate accountability for information quality 
[edit] Data managementData management comprises all the academic disciplines related to managing data as a valuable resource. The official definition provided by DAMA is that "Data Resource Management is the development and execution of architectures, policies, practices and procedures that properly manage the full data lifecycle needs of an enterprise." This definition is fairly broad and encompasses a number of professions which may not have direct technical contact with lower-level aspects of data management, such as relational database management.

[edit] Data miningData mining is the process of sorting through large amounts of data and picking out relevant information. It is usually used by business intelligence organizations, and financial analysts, but is increasingly being used in the sciences to extract information from the enormous data sets generated by modern experimental and observational methods.

It has been described as "the nontrivial extraction of implicit, previously unknown, and potentially useful information from data"[5] and "the science of extracting useful information from large data sets or databases."[6] In relation to enterprise resource planning, according to Monk (2006), data mining is "the statistical and logical analysis of large sets of transaction data, looking for patterns that can aid decision making".[7]

[edit] See alsoData Presentation Architecture 
Scientific visualization 
Software Type Targetted Users License 
Avizo GUI/Code Data Visualisation Engineers and Sciencetists Proprietary 
Data Desk GUI Data Visualisation Statisician Proprietary 
DAVIX Operating System with data tools Security Consultant Various 
Eye-Sys GUI/Code Data Visualisation Engineers and Sciencetists Proprietary 
Ferret Data Visualization and Analysis Gridded Datasets Visualisation Oceanographers and meteorologists Open Source 
Trendalyzer Data Visualisation Teachers Proprietary 
Trendalyzer Specific Data Visualisation Teachers Proprietary 
GGobi GUI Data Visualisation Statisician Open Source 
IBM OpenDX GUI/Code Data Visualisation Engineers and Sciencetists Open Source 
IDL (programming language) Code Data Visualisation Programmer Many 
IDL (programming language) Programming Language Programmer Open Source 
InetSoft Company Many Proprietary 
Instantatlas GIS Data Visualisation Analysts, researchers, statisticians and GIS professionals Proprietary 
MeVisLab GUI/Code Data Visualisation Engineers and Sciencetists Proprietary 
OpenLink AJAX Toolkit Library / Toolkit Programmers GPL 
ParaView GUI/Code Data Visualisation Engineers and Sciencetists BSD 
Processing (programming language) Programming Language Programmers GPL 
Smile (software) GUI/Code Data Visualisation Engineers and Sciencetists Proprietary 
Spotfire GUI Data Visualisation Business Users Proprietary 
StatSoft Company of GUI/Code Data Visualisation Software Engineers and Sciencetists Proprietary 
Tableau Software GUI Data Visualisation Business Users Proprietary 
Trade Space Visualizer GUI/Code Data Visualisation Engineers and Sciencetists Proprietary 
Visifire Library Programmers Open Source 
VisIt GUI/Code Data Visualisation Engineers and Sciencetists Open Source 
VTK C++ Library Programmers Open Source 
Yoix Programming Language Programmers Open Source 
